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About  the  Technical  Note  Series  on  Architecture  Evaluation  in 


the  Department  of  Defense 


The  Product  Line  Systems  Program  is  publishing  a  series  of  technical  notes  designed  to 
condense  knowledge  about  architecture  evaluation  practices  into  a  concise  and  usable  form 
for  Department  of  Defense  (DoD)  acquisition  managers  and  practitioners.  This  series  is  a 
companion  to  the  Software  Engineering  Institute  (SEI)  series  on  product  line  acquisition  and 
business  practices. 

Each  technical  note  in  the  series  focuses  on  the  use  of  architecture  evaluation  and,  in 
particular,  on  applying  the  SETs  architecture  tradeoff  analysis  technology  in  the  DoD  and 
government  organizations.  Our  objective  is  to  provide  practical  guidance  on  how  those 
organizations  can  integrate  sound  architecture  evaluation  practices  into  their  acquisitions. 
This  series  of  technical  notes  lays  down  a  conceptual  foundation  for  DoD  architecture 
evaluation  practice. 
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Abstract 


The  architecture  of  a  software-intensive  system  is  critical  to  its  quality.  For  an  acquisition 
organization  within  the  Department  of  Defense  (DoD),  evaluating  architectures  as  early  as 
possible  in  an  acquisition  can  have  a  favorable  impact  on  the  delivered  system.  This  technical 
note  is  a  case  study  of  how  a  DoD  organization  used  architecture  analysis  and  evaluation  in  a 
major  system  acquisition,  early  on,  to  reduce  program  risk.  The  case  study  begins  by 
describing  the  system,  the  motivation  for  including  architecture  evaluation  in  the  acquisition, 
and  the  Quality  Attribute  Workshop  (QAW)  approach.  Following  this  is  a  brief  description  of 
the  system  acquisition  strategy.  The  case  study  then  describes  the  set  of  events  (and 
supporting  artifacts)  that  were  required  to  incorporate  QAW  architecture  analysis  and 
evaluation  in  the  acquisition  strategy.  In  addition,  it  describes  the  relationship  of  these  events 
and  artifacts  to  the  source-selection  process.  Concluding  the  case  study  is  a  description  of  the 
accomplishments  and  lessons  learned,  along  with  sample  sections  from  the  request  for 
proposal  (RFP).  These  sections  provide  additional  insight  into  the  contractual  language  that 
was  used  to  implement  the  architecture  analysis  and  evaluation  approach. 
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1  Introduction 


This  technical  note  is  a  case  study  of  how  a  Department  of  Defense  (DoD)  organization 
is  applying  architecture  analysis  and  evaluation  in  a  system  acquisition,  early  in  the 
source-selection  process  to  reduce  program  risk. 

In  Sections  1  through  3  of  the  case  study,  we  describe  the  system  being  acquired,  the 
motivation  for  including  architecture  analysis  and  evaluation  in  the  acquisition,  and  the 
Quality  Attribute  Workshop  (QAW)  approach.  Following  this,  we  provide  a  brief 
description  of  the  system  acquisition  strategy  (Section  4)  and  the  set  of  events  and 
supporting  artifacts  that  were  required  to  incorporate  QAW  architecture  analysis  in  the 
acquisition  strategy  (Section  5).  The  relationship  of  these  events  and  artifacts  to  the 
source-selection  process  is  also  described.  Concluding  the  case  study  is  a  description  of 
the  accomplishments  and  lessons  learned  (Section  6),  along  with  sample  sections  from 
the  request  for  proposal  (RFP).  These  sections  provide  additional  insight  into  the 
contractual  language  that  was  used  to  implement  the  architecture  analysis  and  evaluation 
approach. 

In  a  software-intensive  system,  the  architecture  of  the  system  significantly  influences  the 
performance  and  other  qualities  of  that  system.  The  early  use  of  architecture  analysis  and 
evaluation  can  help  mitigate  many  of  the  risks  associated  with  system  development, 
thereby  improving  the  ability  of  an  organization  to  achieve  its  stated  system  objectives.^ 
In  an  acquisition  context,  these  analyses  and  evaluations  provide  the  acquirer  with  a 
proactive  means  of 

•  gaining  early  visibility  into  critical  design  decisions  that  will  drive  the  entire  system 
and  software  development  effort 

•  determining  if  a  system  being  proposed  by  a  supplier  will  satisfy  its  desired  system 
quality  attributes  (for  example,  performance,  availability,  and  modifiability)  before 
the  system  and  software  are  actually  built 

This  is  different  from  an  architectural  review  that  is  a  typical  part  of  an  acquisition 
milestone,  such  as  a  Critical  Design  Review,  These  architectural  reviews  are  relatively 
superficial  and  rarely  address  the  software  architecture^  of  the  system. 


*  Fisher,  M.  “Software  Architecture  Awareness  and  Training  for  Software  Practitioners.”  Written 
for  the  U.S.  Army  CECOM.  Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon 
University,  1998. 

^  The  software  architecture  of  a  program  or  computer  system  is  the  structure  or  structures  of  the 
system,  which  comprise  software  components,  the  externally  visible  properties  of  those 
components,  and  the  relationships  among  them  [Bass  98]. 
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1.1  Terminology 

Source  selection  encompasses  the  process  of  evaluating  offerors  (i.e.,  bidders)  in 
accordance  with  the  evaluation  factors  specified  in  the  governing  RFP  and  awarding  one 
or  more  contracts.  It  is  often  referred  to  as  a  “down  select”  because  it  typically  results  in 
the  elimination  of  many  offerors  from  consideration  and  the  selection  of  a  small  number 
(e.g.,  one  or  two)  for  a  contract  award. 

In  the  DoD  acquisition  environment,  the  term  evaluation  has  special  meaning  because 
evaluation  is  an  integral  part  of  the  source-selection  process  that  is  common  to  all 
government  acquisitions.  What  is  commonly  referred  to  as  an  architecture  evaluation  in 
the  technical  arena  will  be  referred  to  as  an  architecture  analysis  in  this  technical  note. 

We  will  reserve  the  use  of  the  word  evaluation  in  reference  to  the  source-selection 
process.^  In  this  context  then,  conducting  an  architecture  analysis  and  evaluation  means 
analyzing  an  architecture  (and  producing  a  report  on  the  analysis  results)  and  evaluating 
the  analysis  results  in  strict  accordance  with  the  technical  evaluation  criteria  of  Section  M 
of  the  governing  RFP. 


^  Once  source  selection  is  complete  and  a  final  contract  is  awarded,  the  term  evaluation  also 
applies  to  evaluating  the  contractor’s  performance  and  contractual  deliverables  in  accordance 
with  the  specific  provisions  of  the  contract. 


2 


CMU/SEI-2002-TN-013 


2  DoD  System  Acquisition  Context 


This  case  study  involves  an  ongoing  DoD  acquisition.  The  identities  of  the  DoD 
organization  and  system  have  been  purposefully  disguised  because  the  acquisition  is  in  a 
sensitive  phase.  We  will  refer  to  the  DoD  acquisition  organization  as  the  DAC  and  the 
system  they  are  acquiring  as  the  MSIS — a  maintenance  support  information  system. 


2.1  The  System  Being  Acquired:  MSiS 

The  MSIS  is  a  complex  system  of  systems  that  consists  of  three  major  nodes 
encompassing  command,  regional,  and  local  maintenance  centers.  These  centers  support 
the  maintenance  of  weapons  platforms  that  are  operationally  deployed.  Maintaining 
equipment  for  these  platforms  at  the  local  centers  is  performed  by  the  operational  units 
and  is  largely  unscheduled.  Maintenance  at  the  regional  centers  is  performed  on 
platforms  sent  to  the  regional  centers  for  scheduled  overhauls.  Maintenance  at  the 
command  centers,  on  the  other  hand,  focuses  on  maintenance  planning  and  analysis 
activities.  From  a  total  system  perspective,  all  the  centers  have  areas  of  overlapping 
operations  and  functionality.  Figure  1  provides  an  overview  of  the  MSIS  concept. 
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Figure  1.  Conceptual  View  of  the  MSIS 
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The  MSIS  will  have  to  provide  timely  information  to  support  the  maintenance  needs  of  a 
wide  variety  of  DoD  operational  weapons  platforms  at  geographically  dispersed  sites. 

The  MSIS’s  mission  is  to  provide  the  support  needed  by  the  command,  regional,  and 
local  centers  to  plan,  coordinate,  schedule,  and  monitor  the  repair,  maintenance,  and 
upgrade  of  these  weapons  platforms,  or  their  overhaul  and  refurbishment.  For  each 
platform,  the  system  maintains  its  configuration  and  the  part  number  of  each  of  its 
elements.  The  system  also  requisitions  parts  from  inventory  and  maintains  the  data 
associated  with  the  deficiencies  being  repaired  by  the  maintainers  and  the  required 
maintenance  equipment  and  facilities.  The  ability  to  communicate  reliably  with  these 
centers  and  rapidly  access  their  maintenance  information  system  databases  will  be  a 
major  element  in  managing  and  rendering  the  maintenance  services  needed  to  support  the 
MSIS’s  customers.  As  a  result,  the  system  specification  for  the  MSIS  emphasizes  not 
only  functionality  but  also  specific  quality  attributes  (e.g.,  performance,  interoperability, 
availability,  security,  usability,  and  modifiability)  that  reflect  these  needed  system 
capabilities.  Of  course,  the  DAC  must  also  consider  the  functional  requirements 
associated  with  performing  the  MSIS’s  mission,  but  that  is  not  the  focus  of  this  technical 
note.  If  functionality  were  all  that  mattered,  any  monolithic  architecture  would  do,  but 
other  things — namely  the  quality  attributes — ^also  matter.  Throughout  the  development 
process,  the  architecture  must  play  a  role  that  is  both  prescriptive  and  descriptive.  Even  in 
an  incremental  or  spiral  approach,  the  core  architectural  decisions  that  support  the 
important  system  quality  attributes  must  come  first,  and  then  they  can  be  enhanced  in 
future  increments  or  spirals.  An  architecture-centric  approach,  though,  is  key  to  the 
development  of  systems  to  meet  both  their  functionality  and  quality  goals. 

2.2  Motivation  for  Incorporating  Architecture  Analysis 

The  architecture  is  the  foundation  for  any  system.  It  represents  the  earliest  design 
decisions  that  are  both  the  most  difficult  to  get  right  and  the  hardest  to  change 
downstream.  The  architecture  will  allow  or  preclude  nearly  all  of  the  system’s  qualities. 
Modifiability,  performance  predictability,  security,  availability,  interoperability,  and 
usability  are  all  precast  when  the  architecture  has  been  established.  No  amount  of  later 
tuning  and  implementation  tactics  will  compensate  for  the  ills  of  a  poorly  constructed 
architecture.  Experience  has  shown  that  an  unsuitable  architecture  will  eventually 
precipitate  some  sort  of  disaster  on  a  project  [Kazman  00,  Clements  02a].  That  disaster 
may  mean  failure  to  meet  the  performance  goals,  failure  to  interoperate  as  needed,  and/or 
inordinate  sustainment  costs,  among  others.  It  follows  then  that  the  design  of  a  system’s 
architecture  is  key  to  achieving  the  system’s  goals. 

As  a  result,  the  ability  to  analyze  and  evaluate  architectures  early  in  the  acquisition  cycle 
can  help  ensure  that  the  delivered  systems  will  meet  these  goals.  This  was  a  major  driver 
for  incorporating  architecture  analysis  in  the  MSIS  acquisition  and  including  it  as  a  major 
evaluation  factor  in  the  source-selection  process. 
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One  of  the  challenges  facing  the  DAC  was  determining  what  would  be  required  to 
incorporate  architecture  analysis  and  evaluation  in  its  acquisition  so  it  could  analyze  a 
contractor’s  proposed  design  to  see  if  it  satisfied  the  system’s  quality  requirements. 
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3  Quality  Attribute  Workshop  (QAW) 


The  DAC  chose  to  use  the  Software  Engineering  Institute’s  (SEFs)  Quality  Attribute 
Workshop  (QAW)  [Barbacci  02]  as  its  architecture  analysis  “method  of  choice.”  The 
QAW  is  based  on  techniques  successfully  applied  in  the  Architecture  Tradeoff  Analysis 
Method®”  (ATAM®”)  [Kazman  00].  The  purpose  of  the  ATAM  is  to  assess  the 
consequences  of  architectural  decisions  in  light  of  quality  attribute  requirements  and 
business  drivers.  Not  only  can  it  analyze  specific  architecture  quality  attributes,  but  it 
also  allows  engineering  tradeoffs  to  be  made  among  possibly  conflicting  system  quality 
goals.  In  this  way,  an  ATAM  evaluation  can  detect  areas  of  potential  risk  within  the 
software  architecture  of  a  complex,  software-intensive  system.  Clements  and  associates 
provide  details  and  uses  of  the  ATAM  (and  other  software  architecture  analysis  methods) 
[Clements  02a].  The  software  architecture  must  be  documented  before  the  ATAM  can  be 
conducted.  One  of  the  steps  of  the  ATAM  involves  facilitating  a  group  of  system 
stakeholders  to  brainstorm  and  prioritize  scenarios  that  characterize  required  quality 
attributes.  The  architecture  is  then  analyzed  against  these  scenarios.  The  results  of  an 
ATAM  evaluation  include  a  listing  of  risks,  tradeoffs,  and  sensitivity  points. 

In  the  QAW,  the  quality  attributes  and  business  drivers  are  also  established  early,  and  the 
same  technique  that  is  used  in  ATAM  is  used  to  brainstorm  and  prioritize  scenarios. 

These  scenarios  are  then  turned  into  architectural  test  cases  (ATCs),  which  also  include 
questions  concerning  the  architectural  issues  related  to  the  desired  quality  attributes  and 
suggestions  for  how  to  respond  to  each  question.  The  system  developer  is  responsible  for 
building  the  architecture  and  performing  the  entire  analysis  of  the  architecture  offline, 
while  the  role  of  the  architecture  evaluation  team  is  to  review  and  evaluate  the  analysis 
results.  The  QAW  method  is  an  outgrowth  of  SEI  work  with  DoD  customers  who  wanted 
to  apply  architecture  analysis  and  evaluation  early  in  the  acquisition  cycle.  The  method 
itself  is  still  evolving  as  we  learn  to  wrestle  with  the  problem  of  applying  architecture 
analysis  before  a  software  architecture  has  been  fully  crafted. 

In  the  QAW,  unlike  in  the  ATAM,  the  architecture  need  not  be  available  at  the  early 
stages  when  the  scenarios  are  generated  and  the  ATCs  are  built.  However,  at  least  a 
partial  architecture  must  be  available  before  the  analysis  of  the  test  cases  can  proceed. 

The  architecture  must  be  sufficiently  detailed  to  satisfy  the  “expected  response”  for  each 
“question”  in  each  ATC,  and  hence  will  include  elements  of  both  a  system  architecture 
[AWG  98]  and  a  software  architecture  [Clements  02a].  It  is  fully  expected  that  the  ATCs 
will  drive  the  early  architecture  view  development,  since  the  questions  define  which  parts 
of  the  architecture  will  be  analyzed,  and  the  expected  responses  indicate  the  level  of 


Architecture  Tradeoff  Analysis  Method  and  ATAM  are  service  marks  of  Carnegie  Mellon 
University. 
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detail  necessary  in  the  analysis.  Hence,  the  sequence  diagrams  built  in  the  early  stages 
will  be  those  that  are  called  out  in  an  “expected  response”  section.  This  is  appropriate, 
since  the  ATCs  represent  high-priority  operational  cases  that  strongly  correspond  to  the 
project’s  business  drivers. 

Like  the  ATAM,  the  QAW  helps  to  uncover  risks  related  to  architectural  decisions  that 
might  create  future  problems  with  regard  to  a  quality  attribute  goal.  Discovered  risks  can 
then  be  evaluated  and  made  the  focus  of  mitigation  activities  (e.g.,  further  design, 
analysis,  or  prototyping).  The  QAW  provides  an  early  reasoning  framework  that  can 
guide  system  development  to  help  ensure  that  quality  goals  are  achieved. 


3.1  QAW  Process 

In  this  section,  we  provide  a  brief  overview  of  the  QAW  process  and  leave  it  to  the 
readers  to  familiarize  themselves  with  all  the  technical  details  of  the  QAW  process 
[Barbacci  02].  The  QAW  process  consists  of  four  distinct  groups  of  activities  shown  in 
Figure  2.  These  activities  include:  (1)  Scenario  Generation,  (2)  Test  Case  Development, 
(3)  Test  Case  Architecture  Analysis,  and  (4)  Analysis  Results  Presentation. 


Figure  2.  Overview  of  the  QAW  Process 

3.1 .1  Scenario  Generation 

A  scenario  is  a  short  statement  describing  an  interaction  of  one  or  more  stakeholders  with 
the  system  [Clements  02a].  Scenarios  are  used  to  represent  stakeholders’  interests  and 
quality  attribute  requirements  and  to  exercise  the  architecture  against  current  and  future 
situations.  During  scenario  generation,  individual  stakeholders,  in  a  round-robin 
brainstorming  fashion,  propose  scenarios  or  ask  questions  about  the  way  in  which  the 
architecture  will  respond  to  various  situations.  The  stakeholders  who  typically  take  part 
in  this  QAW  activity  include  domain  experts,  technology  experts,  maintainers,  and  users. 


CMU/SEI-2002-TN-013 


7 


The  output  of  this  activity  is  a  prioritized  set  of  scenarios  with  an  additional  refinement 
of  the  highest  priority  scenarios  (typically  3  to  5).  Appendix  A  contains  an  example  of  a 
typical  MSIS  scenario. 

3.1 .2  Test  Case  Development 

The  same  stakeholders  who  participate  in  scenario  generation  are  usually  involved  in  test 
case  development.  Test  case  development  transforms  each  refined  scenario  into  a  well- 
documented  test  case.  A  test  case  is  a  fully  developed,  robust  scenario  that  includes 

•  a  context  section  that  outlines  the  operational  conditions  that  form  the  basis  for  the 
test  case 

•  a  series  of  questions  stating  the  corresponding  architectural  issues  and  concerns 

•  an  expected  response  (for  each  question)  that  suggests  how  the  architecture 
development  team  should  respond  to  each  question 

•  a  utility  table'*  that  summarizes  the  quality  attributes,  the  particular  system  aspects 
being  addressed,  and  their  relationship  to  the  questions 

Appendix  A  contains  an  example  ATC  for  MSIS  and  its  corresponding  scenario. 

3.1 .3  Test  Case  Architecture  Analysis 

In  the  Test  Case  Architecture  Analysis  activity,  the  architecture  development  team 
independently  analyzes  the  architecture  using  the  ATCs  and  documents  the  results.  An 
appropriate  set  of  ATCs  and  architecture  documentation  are  prerequisites  for  performing 
this  analysis.  Because  there  are  no  generally  accepted,  industry-wide  standards  for 
describing  architectures,  the  analyses  are  often  constrained  by  the  available 
documentation.  In  the  case  of  systems  documented  using  the  Command,  Control, 
Communications,  Computer,  Intelligence,  Surveillance  and  Reconnaissance  (C4ISR) 
Architecture  Framework^  [AWG  98],  different  products  or  collections  of  products  will 
differ  in  their  relative  value  for  analyzing  quality-attribute-specific  ATCs  [Barbacci  99]. 
Depending  on  the  quality  attributes  of  concern,  C4ISR  products  will  have  to  be 
augmented  with  additional  architectural  views  and  documentation  to  address  quality 
attribute  concerns  that  are  under-represented  in  the  C4ISR  products.  For  example,  the 
architecture  development  team  may  need  sequence  diagrams  showing  the  behavior  of  the 
major  system  components  and  the  sequences  of  messages  passing  between  them. 
Clements  and  associates  provide  practical  guidance  on  how  to  capture  an  architecture  in 
written  form  so  it  can  fulfill  its  purpose  as  a  communication  vehicle  for  all  the  varied 
stakeholders  in  a  system’s  development  [Clements  02b]. 


■*  The  summation  of  all  the  utility  table  entries  for  all  the  test  cases  represents  a  utility  tree  for 
the  architecture  analysis  and  evaluation.  See  page  32  for  an  example  of  a  utility  tree. 

^  The  C4ISR  Architecture  Framework  [AWG  98]  is  becoming  the  required  method  for 
describing  information  system  architectures  within  the  DoD  and  other  U.S.  government 
agencies. 
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3.1 .4  Analysis  Results  Presentation 

The  Analysis  Results  Presentation  is  the  final  activity  in  the  QAW  process.  It  is  a  one-  or 
two-day  meeting  attended  by  the  architecture  evaluation  team  and  the  architecture 
development  team.  It  provides  an  opportunity  for  the  architecture  development  team  to 
present  the  results  of  its  analysis  and  to  demonstrate  that  the  proposed  architecture  can 
handle  the  ATCs  correctly.  A  workbook  containing  a  summary  of  the  QAW  process,  the 
collection  of  ATCs,  and  example  analyses  and  results  presentations  is  provided  to  the 
architecture  development  team  in  advance.  The  workbook  contains  example  artifacts  that 
are  useful  for  guiding  the  team  through  each  QAW  activity. 

3.2  Applying  the  QAW  Process 

Using  the  QAW  process  in  a  system  acquisition  context  requires  careful  planning  that 
involves  appropriately  tailoring  and  integrating  the  four  QAW  activities  in  a  manner  that 
is  fully  compatible  with  the  needs  of  the  acquiring  organization.  For  the  MSIS,  the 
starting  point  was  to  understand  the  DAC’s  acquisition  strategy.  This  is  the  focus  of  the 
next  section. 
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4  The  Acquisition  Strategy  of  the  MSIS 


The  acquirer’s  acquisition  strategy  defines  its  overall  approach  for  how  it  intends  to 
acquire  a  system.  Figure  3  depicts  the  five  phases  in  the  overarching  acquisition  strategy 
the  DAC  selected  for  the  MSIS  acquisition.  A  substantial  Planning  and  Preparation  phase 
(during  which  the  acquisition  strategy  was  developed)  preceded  these  five  phases. 


Competitive 

Elicitation 


Bidders’ 

Conferences 


Initial  Competitive  Final  System 

Dpvim  Elect  Fly-Off  Down  Select  Implementation 

I - - - - ►!— - ^ ^ 


Figure  3.  Overview  of  the  DAC's  Acquisition  Strategy  for  Acquiring  the  MSIS 

The  strategy  begins  with  an  “open”  or  competitive  solicitation,  followed  by  a  source 
selection  that  leads  to  an  initial  down  select.  This  initial  down  select  results  in  contract 
awards  to  two  suppliers  to  participate  in  a  competitive  system  development  commonly 
referred  to  as  a  “fly-off.”  The  selected  suppliers  (contractors  A  and  B)  subsequently  begin 
the  Competitive  Fly-Off — the  initial  performance  phase  of  their  MSIS  development 
effort — in  accordance  with  their  technical  proposal/contract.  Near  the  end  of  the  fly-off, 
the  DAC  issues  a  Call  for  Improvement  (CFI)  requesting  each  contractor  to  submit  a 
revised  technical  proposal  that  incorporates  the  results  of  the  work  they  performed  and 
the  understanding  they  gained  during  the  competitive  fly-off.  In  the  final  down  select, 
the  acquirer  evaluates  each  contractor’s  improved  technical  proposals,  and  awards  a 
contract  for  system  implementation  to  the  supplier  submitting  the  “best  value”  proposal. 
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An  important  aspect  of  the  acquirer’s  acquisition  strategy  is  that  it  sets  the  context  for 
incorporating  architecture  analysis  and  evaluation  in  an  acquisition.  The  next  section 
discusses  the  planning  and  preparation  that  went  into  integrating  the  QAW  with  the  MSIS 
acquisition  strategy. 
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5  Integrating  the  QAW  in  the  MSIS  Acquisition 


The  acquisition  phases  (depicted  in  Figure  4)  serve  as  a  roadmap  to  show  how  the  SEI 
assisted  the  DAC  in  incorporating  the  QAW  into  its  acquisition.  This  roadmap 
corresponds  to  the  initial  acquisition  Planning  and  Preparation  phase  followed  by  the  five 
phases  of  the  acquisition  strategy. 

Planning  and  pompetitjve  initial  Competitive  Final  System 

Preparation  Sdlicli^iion  Down  Select  Fly-Off  Down  Select  Implementation 

Figure  4.  MSIS  Acquisition  Roadmap 

The  subsections  that  follow  correspond  one  to  one  with  these  six  phases  and  include 
tables  that  describe  the  specific  events  and  supporting  artifacts  that  were  required  to 
create  the  acquisition  infrastructure.  This  infrastructure  accommodates  the  QAW  process 
and  provides  a  suitable  means  for  evaluating  the  QAW  architecture  analysis  results.  Table 
1  provides  the  legend  for  understanding  the  allocation  of  roles  and  responsibilities  for  the 
numbered  events  described  in  the  other  tables  in  this  section. 


BLACK  FILL  around  the  event  number  means  that  the  DAC  is  responsible 
for  performing  all  the  tasks  for  that  particular  event. 


DARK  GRAY  FILL  around  the  event  number  means  the  DAC  and  the 
contractor  are  jointly  responsible  for  conducting  the  event  in  a  collaborative 
manner  (in  accordance  with  the  governing  plan). 

Aliy  task  that  is  SHADED  IN  LIGHT  GRAY  in  the  joint  event  is  performed 
by  the  DAG. 

Any  task  that  is  NOT  SHADED  in  the  joint  event  is  performed  by  the 
contractor.  _ 

NO  FILL  around  the  event  number  means  that  the  offeror  or  contractor  is 
responsible  for  performing  all  the  tasks  for  that  event  (in  accordance  with  the 
RFP  or  an  approved  plan). 


Table  1.  Legend  for  Identifying  Roles  and  Responsibilities  for  QAW  Events 

5.1  Planning  and  Preparation 

Typical  of  acquisitions  of  this  magnitude,  the  MSIS  acquisition  involved  a  significant 
amount  of  planning  and  preparation.  This  work  involved  establishing  the  acquisition 
objectives,  developing  the  detailed  acquisition  strategy,  identifying  and  managing  risks, 
specifying  the  system  requirements,  developing  cost  and  schedule  estimates,  and 
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preparing  the  RFP.  The  DAC  was  responsible  for  executing  and  coordinating  all  of  these 
activities. 

A  major  way  in  which  the  DAC  elected  to  reduce  overall  program  risk  was  to  incorporate 
QAW-based  architecture  analysis  and  evaluation  and  to  integrate  it  into  the  DAC’s 
source-selection  process.  This  required  additional  planning  and  preparation.  Table  2 
describes  the  acquisition  Planning  and  Preparation  phase  events  that  are  relevant  to 
integrating  the  QAW  in  the  MSIS  acquisition. 


PHASE 

EVENT 

Description  of  Event  and  Relevance  to  QAW 

DAC  management  establishes  the  objectives  and  governing  policies  and 
parameters  for  the  MSIS  acquisition.  Some  of  these  affect  how  the  QAW  was 
integrated  into  the  acquisition.  Examples  include 

•  mandating  compliance  with  the  prescribed  acquisition  schedule 

•  mandating  the  use  of  a  Statement  of  Objectives  (SOO)  approach  for 
structuring  the  RFP  instead  of  a  traditional  Statement  of  Work  (SOW) 

•  requiring  some  aspect  of  the  QAW  to  serve  as  a  “discriminator”  in 
source  selection  for  both  the  initial  and  final  down  select 

Z 

o 

The  DAC  forms  a  team  to  plan  the  acquisition,  develop  an  acquisition 

strategy,  and  prepare  the  RFP.  In  conjunction  with  the  DAC,  SEI  team 

oc 

members 

•  identify  the  business  drivers  and  system  quality  attributes  that  are 

UJ 

cr 

important  and  that  reflect  the  business  drivers 

Q. 

•  plan  and  carry  out  the  first  two  QAW  activities — Scenario  Generation 

Q 

and  Architecture  Test  Case  Development — in  parallel  with  the  other 

< 

acquisition  planning  and  preparation  efforts 

(D 

•  coordinate  the  specification  of  quality  attribute  requirements 

•  coordinate  the  specification  of  the  architectural  documentation 

Z 

z 

requirements  (e.g.,  augmented  C4ISR  views)  for  analyzing  the 

< 

1 

architecture’s  ability  to  support  the  quality  attribute  requirements 

Q. 

•  specify  appropriate  acquisition  artifacts  and  develop  corresponding 
language  for  the  SOO  and  Sections  H,  L,  and  M  of  the  RFP  to  integrate 
the  remaining  QAW  activities — ^Test  Case  Architecture  Analysis  and 
Analysis  Results  Presentation — consistent  with  the  acquisition  events 
described  in  these  tables 

The  DAC  completes  the  RFP  and  obtains  management  approval  to  begin  the 
solicitation; 

•  included  in  the  RFP  is  a  set  of  robust  ATCs  for  use  in  the  QAW  Test 

Case  Architecture  Analysis  and  Analysis  Results  Presentation  activities 

Table  2.  Acquisition  Planning  and  Preparation — QAW-Related  Events 
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As  part  of  the  acquisition  Planning  and  Preparation  phase,  SEI  team  members  were 
responsible  for  carrying  out  the  first  two  QAW  activities — Scenario  Generation  and  Test 
Case  Development.  This  involved  planning  and  facilitating  four  separate  working 
sessions  with  MSIS  stakeholders  at  three  different  sites  over  a  period  of  several  months. 
These  sessions  included  stakeholders  from  the  command,  regional,  and  local  centers.  The 
stakeholders  included  acquirers,  domain  experts,  and  prospective  users  of  the  MSIS. 

As  part  of  performing  these  two  QAW  activities,  the  DAC/SEI  team  identified 

•  the  business  drivers  for  the  MSIS 

•  the  system  quality  attributes  that  are  important  and  that  reflect  the  business  drivers 

•  architectural  documentation  (e.g.,  augmented  C4ISR  views)  that  are  needed  to  permit 
an  analysis  of  the  architecture’s  ability  to  support  the  desired  quality  attributes 

The  end  goal  was  to  quickly  develop  a  sufficient  number  of  ATCs  in  time  to  be  included 
in  the  RFP.  These  ATCs  reflected  the  quality  attributes  that  were  essential  to  conducting  a 
core  set  of  maintenance  tasks  at  the  command,  regional,  and  local  centers.  In  developing 
these  ATCs,  there  was  a  strong  emphasis  on  the  processes  and  personnel  used  to  perform 
these  maintenance  tasks  over  a  long  period  of  time,  which  required  extensive  context 
setting. 

All  together,  eleven  ATCs  were  generated  for  inclusion  in  the  RFP.  They  will  drive  the 
final  two  QAW  activities — ^Test  Case  Architecture  Analysis  and  Analysis  Results 
Presentation.  Performing  these  remaining  two  QAW  activities  is  the  responsibility  of  the 
contractors  who  win  the  MSIS  competitive  fly-off  contracts.  How  the  MSIS  contractors 
are  expected  to  conduct  these  activities  (in  conjunction  with  the  DAC)  is  described  in 
Section  5.4. 

5.2  Competitive  Soiicitation 

The  second  acquisition  phase  is  the  Competitive  Solicitation  that  encompasses  three 
events  relevant  to  integrating  the  QAW  in  the  MSIS  acquisition.  Table  3  describes  them. 
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PHASE 

EVENT 

Description  of  Event  and  Relevance  to  QAW 

Z 

o 

i 

O 

-J 

o 

1. 

The  DAC  conducts  bidders’  conferences  to  inform  potential  offerors  about 
the  particulars  of  the  upcoming  MSIS  acquisition. 

•  The  SEI  in  conjunction  with  the  DAC  presents  briefing  on  the  QAW 
process  to  familiarize  offerors  with  the  architecture  analysis  approach 
and  acquisition  implications. 

2. 

The  DAC  releases  the  RFP: 

•  ATCs  are  included  in  the  RFP  as  government-furnished  items  (GFIs)  so 

LU 

that  offerors  can  gauge  the  scope  of  the  required  analysis. 

t 

H 

•  QAW  workbooks  are  also  included  in  the  RFP  to  serve  as  a  guide  for 

planning  and  conducting  the  QAW  Test  Case  Architecture  Analysis  and 

UJ 

Q. 

Analysis  Results  Presentation  activities. 

s 

o 

3. 

Offerors  deliver  technical  proposals  in  accordance  with  Section  L  of  the  RFP: 

o 

•  Section  L  requires  that  the  technical  proposals  include  an  initial  . 
architecture  analysis  plan  (AAP). 

Table  3.  Competitive  Solicitation — QA  W -Related  Events 

A  key  aspect  of  this  phase  is  requiring  each  offeror^  to  submit  an  initial  architecture 
analysis  plan  (AAP)  as  part  of  its  technical  proposal.  The  AAP  describes  how  the  offeror 
intends  to  conduct  QAW-based  architecture  analysis  if  it  is  awarded  a  contract  to 
participate  in  the  competitive  fly^off.  The  scope  of  the  plan  is  to  include  both  the  QAW 
Test  Case  Architecture  Analysis  and  Analysis  Results  Presentation  activities.  Since  each 
offeror  is  required  to  submit  an  initial  plan,  the  DAC  will  have  a  tangible  means  for 
evaluating  each  offeror’s  ability  to 

•  understand  the  QAW  process  and  plan  a  QAW-based  architecture  analysis 

•  make  reasonable  assumptions  in  applying  the  QAW  process  and  integrate  it 
appropriately  into  the  offeror’s  proposed  development  effort 

•  satisfy  the  technical  proposal  requirements  of  the  RFP  that  pertain  to  QAW 
architecture  analysis 

This  approach  allows  architecture  analysis  to  be  a  factor  in  the  initial  source  selection  and 
greatly  reduces  the  risk  of  selecting  an  offeror  that  will  be  unable  to  follow  through 
successfully. 


^  The  term  offeror  refers  to  any  bidder  who  submits  a  technical  proposal.  The  term  contractor  is 
used  to  distinguish  between  a  general  offeror  and  the  two  offerors  who  are  selected  and 
awarded  a  contract  to  participate  in  the  competitive  fly-off  and  final  down  select. 
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5.3  Initial  Down  Select 

The  third  acquisition  phase  is  the  Initial  Down  Select  that  encompasses  three  events 
relevant  to  integrating  the  QAW  in  the  MSIS  acquisition.  Table  4  describes  them. 


PHASE 

EVENT 

Description  of  Event  and  Relevance  to  QAW 

1. 

The  DAC  evaluates  offerors’  technical  proposals  in  accordance  with  the 
technical  evaluation  criteria  in  Section  M  of  the  RFP  (for  the  initial  down 

1- 

select) 

O 

UJ 

•  This  includes  evaluating  each  offeror’s  initial  AAP. 

_J 

lU 

0} 

2. 

The  DAC  factors  in  the  results  of  the  technical  proposal  evaluations  in  source 
selection  in  accordance  with  the  overall  weighting  criteria  in  Section  M  of 

z 

the  RFP  (for  initial  down  select): 

o 

•  AAP  evaluation  results  are  a  significant  factor  in  the  evaluation  of  the 

Q 

■ 

technical  proposals. 

< 

3. 

The  DAC  makes  the  initial  down  select  and  awards  contracts  to  two  of  the 

H 

z 

offerors  to  participate  in  the  competitive  fly-off  in  accordance  with  the 

MM 

DAC’s  source-selection  plan.’ 

•  Each  selected  offeror  will  be  responsible  for  conducting  QAW  activities 
in  accordance  with  its  plan  and  the  RFP/contract  requirements. 

Table  4.  Initial  Down  Select — QAW-Related  Events 


The  DAC  anticipated  receiving  many  technical  proposals  prior  to  the  initial  down  select 
and  needed  an  effective  means  to  screen  out  all  but  the  most  highly  qualified  offerors  in 
order  to  minimize  program  risk  and  make  the  down  select  more  efficient.  Moreover,  due 
to  the  importance  of  architecture  analysis  to  the  MSIS  program,  the  DAC  wanted  some 
aspect  of  architecture  analysis  and  evaluation  to  play  a  major  role  in  source  selection — 
even  in  the  initial  down  select.  The  means  of  achieving  this  is  to  evaluate  the  efficacy  of 
each  offeror’s  AAP  submitted  as  part  of  its  technical  proposal  during  the  initial  down 
select.  Otherwise,  architecture  analysis  and  evaluation  could  not  play  a  role  in  source 
selection  until  the  final  down  select  when  the  analysis  results  from  the  Competitive  Fly- 
Off  phase  would  come  into  play. 

5.4  Competitive  Fly-Off 

The  last  two  QAW  activities — ^Test  Case  Architecture  Analysis  and  Analysis  Results 
Presentation — take  place  in  the  Competitive  Fly-Off  phase.  The  contractors  conduct  these 
activities  in  two  segments;  doing  so  benefits  both  the  DAC  and  the  contractors  as 
explained  in  the  following  sections.  In  the  first  segment,  the  contractors  conduct  a  “dry- 


^  The  source-selection  plan  describes  how  the  DAC  intends  to  evaluate  offerors,  determine 
which  contractors  have  the  best  value  proposals,  and  award  contracts. 
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run”  Test  Case  Architecture  Analysis  and  Analysis  Results  Presentation.  In  the  second 
segment,  they  conduct  “full-scale”  ones.  The  main  difference  between  these  two 
segments  is  that  the  dry-run  segment  involves  the  use  of  only  a  few  designated  ATCs, 
while  the  full-scale  segment  requires  the  use  of  all  the  GFI  ATCs. 

5.4.1  QAW  Dry-Run  Segment 

The  dry-run  segment  encompasses  the  seven  events  described  in  Table  5.  The  benefit  of 
first  having  a  dry  run  is  that  it  provides  the  contractors  with  an  opportunity  to  familiarize 
themselves  with  the  QAW  process,  fine-tune  their  analysis  skills,  make  any  needed 
architectural  changes,  and  refine  their  plans  for  performing  the  analysis. 

5.4.2  QAW  Full-Scale  Segment 

Table  6  describes  the  five  events  encompassing  the  full-scale  segment.  This  segment  of 
the  competitive  fly-off  involves  all  the  ATCs  and  requires  a  final  report  of  the  analysis 
results.  A  unique  aspect  of  this  segment  is  that  the  DAC  will  issue  a  CFI 30  days  before 
the  segment  is  scheduled  to  end.  In  response  to  the  CFI,  the  contractors  must  prepare  an 
improved  technical  proposal  describing  their  plans  for  the  final  System  Implementation 
phase.  Since  the  final  architecture  analysis  report  is  required  to  be  part  of  the  improved 
technical  proposal,  the  analysis  results  can  be  included  as  a  major  evaluation  factor  in 
source  selection  for  the  final  down  select. 
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PHASE 

EVENT 

Description  of  Event  and  Relevance  to  QAW 

The  DAC  conducts  a  kick-off  briefing  with  each  of  the  two  contractors  to  review 
government  expectations  for  the  competitive  fly-off. 

•  The  contracting  officer  reviews  the  rules  of  engagement  (ROE).® 

•  The  DAC  reviews  the  requirements  governing  the  fly-off  and  the  SEI 
presents  a  briefing  on  the  QAW  analysis  and  evaluation  approach. 

u. 

u. 

2. 

Contractors  refine  their  initial  AAPs,  as  necessary,  based  on  the  comments  they 
received  from  the  DAC  (originating  from  the  initial  down  select). 

o 

t 

u 

The  DAC  reviews  each  contractor’s  refined  AAP  and  grants  preliminary 
approval  (or  cites  rework  required  by  the  contractor). 

lU 

> 

1- 

b 

a. 

S 

o 

o 

4. 

Upon  receiving  preliminary  approval  of  its  refined  AAP,  each  contractor 

•  builds  sufficiently  detailed  MSIS  architectural  views  and  documentation  to 
enable  the  architecture  analysis  for  the  dry-run  test  cases 

•  performs  a  dry-run  QAW  Test  Case  Architecture  Analysis’  using  designated 
GFI  ATCs  and  available  architectural  views 

•  prepares  a  presentation  detailing  the  architecture  analysis  results 

o 

4-1 

c 

o 

E 

o> 

0) 

<n 

Each  contractor  in  conjunction  with  the  DAC  conducts  a  dry-run  QAW  Analysis 
Results  Presentation.'® 

•  Each  contractor  presents  the  architecture  analysis  results  for  the  small  set  of 
designated  ATCs  on  a  test-case-by-test-case  basis. 

•  The  DAC  reviews  tmd  discusses  the  analysis  results  and  the  contractors’ 
progress  against  their  refined  AAPs. 

•  The  DAC  issues  evaluation  notices  (ENs),"  as  necessary,  based  on  its 
review  and  technical  interchange  with  the  contractors  in  accordance  with 
theQAW’sROE. 

6. 

Each  contractor  revises  its  AAP,  as  necessary,  to  incorporate  planning  changes 
and  refinements,  or  to  accommodate  the  ENs  received  from  the  DAC. 

1 

The  DAC  reviews  each  contractor’s  revised  AAP  and  grants  final  approval  (or 
cites  rework  required  by  the  contractor). 

Table  5.  Competitive  Fly-Off— QA  W-Related  Events  for  Dry-Run  Segment 


®  The  rules  of  engagement  (ROE)  govern  the  administrative  and  technical  interchange  and 
conduct  of  both  the  DAC  and  the  contractors  during  the  competitive  fly-off. 

’  This  task  corresponds  to  Activity  #3  of  the  QAW  process  shown  in  Figure  2  on  page  7 . 

This  task  corresponds  to  Activity  #4  of  the  QAW  process  shown  in  Figure  2  on  page  7 . 

"  The  only  agent  authorized  to  issue  ENs  on  behalf  of  the  DAC’s  architecture  evaluation  team  is 
the  contracting  officer.  The  ENs  identify  items  needing  clarification  or  deficiencies  requiring 
resolution.  Deficiencies  cite  conditions  that  will  not  satisfy  the  system  requirements  (including 
failure  to  meet  the  expected  response  for  test  cases)  or  conditions  that  represent  potential  risks 
in  terms  of  meeting  the  system  quality  requirements. 
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PHASE 

EVENT 

Description  of  Event  and  Relevance  to  QAW 

Second  Segment  of  COMPETITIVE  FLY-OFF 

8. 

Upon  receiving  final  approval  of  its  revised  AAP,  each  contractor 

•  completes  build  of  all  MSIS  architectural  views  and  documentation  to 
enable  the  architecture  analysis  for  the  full-scale  test  cases 

•  performs  a  full-scale  QAW  Test  Case  Architecture  Analysis  using  all 

GFI  ATCs  and  completed  architectural  views 

•  prepares  a  presentation  describing  the  architecture  analysis  results 

'-IK'::-:-'; 

-I;. 

-  vh  , 

Each  contractor  in  conjunction  with  the  DAC  conducts  a  full-scale  QAW 
Analysis  Results  Presentation. 

•  Each  contractor  presents  the  architecture  analysis  results  for  all  ATCs 
on  a  test-case-by-test-case  basis. 

•  The  DAC  reviews  (and  conducts  dialog  on)  the  analysis  results  and 
the  contractors’  progress,  comparing  it  against  their  revised  AAPs. 

•  The  DAC  issues  ENs,  as  necessary,  based  on  its  review  and  technical 
interchange  with  the  contractors. 

10. 

Each  contractor  prepares  an  Architecture  Analysis  Report  (AAR)  in 
accordance  with  Section  L  of  the  RFP.  The  AAR  includes 

•  architecture  analysis  results  that  were  presented  during  the  QAW 
presentation  of  the  analysis  results  for  all  ATCs 

•  architectural  views  and  documentation  used  in  the  analysis  and  other 
supporting  information  as  described  in  QAW  workbooks 

•  responses  to  all  ENs  issued  by  the  DAC  describing  either  how  the  ENs 
were  resolved  or  mitigation  plans  for  resolving  them 

•  any  new  or  revised  architectural  documentation  corresponding  to 
proposed  changes  or  resolution  of  ENs  received  from  the  DAC 

The  DAC  issues  a  CFI. 

12. 

Contractors  deliver  their  improved  technical  proposals,  in  accordance  with 
Section  L  of  the  RFP.  The  improved  proposals  include 

•  their  final  AAR  and  AAP 

•  a  complete  description  of  any  new  architectural  changes  being  proposed 
including  the  supporting  rationale  and  impact  on  architecture  analysis 
results  previously  presented  by  the  contractor 

•  an  improved  AAP  (lAAP)  describing  the  contractor’s  approach  for 
conducting  an  architecture  analysis  of  each  software  architecture  build 
(i.e.,  increment)  using  the  ATAM  during  the  System  Implementation 
phase 

Table  6,  Competitive  Fly-Off—QA  W-Related  Events  for  Full-Scale  Segment 
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FINAL  DOWN  SELECT  PHASE 


5.5  Final  Down  Select 

The  Final  Down  Select  phase  encompasses  five  events  that  are  relevant  to  integrating  the 
QAW.  Table  7  describes  them. 


Description  of  Event  and  Relevance  to  QAW 


The  DAC  evaluates  the  improved  technical  proposals  in  accordance  with 
technical  evaluation  criteria  in  Section  M  of  the  RfP  (for  the  final  down 
select). 

•  This  includes  evaluating  the  AAR,'^  progress  against  the  AAP,  and  the 

lAAP.  _ _ 

Each  contractor  presents  (as  part  of  its  oral  presentation  to  the  DAC)  a 
summary  of  its  architecture  analysis  results  (including  any  subsequent 
architectural  changes  and  their  impact),  overall  performance  and  progress 
compared  against  its  AAP,  and  its  proposed  lAAP. 

•  The  dag  conducts  separate  discussions  with  each  contractor  on  its  oral 
presentations  in  accordance  with  the  ROE. 

The  DAC  evaluates  each  contractor’s  oral  presentation  in  accordance  with 
the  technical  evaluation  criteria  in  Section  M  of  the  RFP  (for  the  final  down 
select). 


The  DAC  incorporates  the  evaluation  results  (encompassing  improved 
technical  proposals  and  oral  presentations)  into  source  selection  in 
accordance  with  the  weighting  criteria  in  Section  M  of  the  RFP  (for  the  final 
down  select). 

•  The  architecture  evaluation  results  are  a  significant  factor  in  the  overall 
evaluation  and  selection  process.  _ 

The  DAC  makes  the  final  down  select  and  awards  a  system  implementation 
contract  to  one  of  the  two  fly-off  contractors  in  accordance  with  the  DAC’s 
source-selection  plan.'^ 


•  The  winning  contractor  is  responsible  for  conducting  follow-on 
software  architecture  analyses  during  the  System  Implementation  phase 
in  accordance  with  its  submitted  plan  and  the  RFP/contract 
requirements.  _ 


Table  7.  Final  Down  Select— QAW-Related  Events 


A  key  aspect  of  this  phase  is  that  the  DAC  architecture  evaluation  team  has  an  opportunity 
to  review  and  discuss  the  final  analysis  report  as  well  as  the  contractor’s  proposed 
architectural  changes  (or  plans)  for  mitigating  discovered  risks  or  deficiencies  that  were 


The  AAR  describes  the  detailed  results  of  the  architecture  analysis  the  contractor  performed 
during  the  competitive  fly-off. 

The  source-selection  plan  must  cover  both  the  initial  down  select  and  the  final  down  select. 
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cited  in  the  ENs.  This  takes  place  during  the  contractor’s  oral  presentation,  which  is  a 
traditional  element  of  the  source-selection  process.  The  contractors  must  submit  their 
analysis  reports  in  advance  of  their  oral  presentations.  This  allows  the  DAC  architecture 
evaluation  team  ample  opportunity  to  carefully  review  the  reports  prior  to  the  oral 
presentations.  These  presentations  are  also  a  factor  in  evaluating  the  performance  of  the 
competing  contractors. 


5.6  System  Implementation 

The  System  Implementation  phase  encompasses  two  follow-on  events  to  the  QAW  that 
are  relevant  to  integrating  architecture  analysis  in  the  MSIS  acquisition.  Table  8  describes 
them. 


PHASE 

EVENT 

Description  of  Event  and  Relevance  to  QAW 

1. 

The  winning  contractor  refines  its  System  Evolution  Plan  (SEP)'''and 
performs  the  following  activities: 

z 

o 

•  The  contractor  refines  its  system  architecture  in  increments 

H: 

corresponding  to  the  architecture’s  approved  SEP. 

? 

•  The  contractor  refines  its  system  architecture‘s  in  increments 

z 

LU 

corresponding  to  its  approved  SEP. 

LU 

•  The  contractor  refines  and  completes  the  development  of  its  software 

architecture  in  increments  corresponding  to  its  approved  SEP. 

S 

•  The  contractor  conducts  an  architecture  analysis  of  each  increment  of 

the  software  architecture  using  the  ATAM  in  accordance  with  its 

S 

LU 

approved  lAAP. 

h- 

<0 

The  DAC  evaluates  the  software  architecture  analysis  results  for  each 

increment  specified  in  the  lAAP. 

•  The  DAC  appropriately  factors  ATAM  evaluation  results  into  an 
incentive  award  in  accordance  with  specific  contract  provisions. 

Table  8.  System  Implementation  Phase — ATAM-Related  Events 


A  key  aspect  of  this  phase  is  that  the  implementation  contractor  will  conduct  software 
architecture  evaluations  using  the  ATAM.  These  in-situ  architecture  evaluations  will  assist 
the  DAC  in  evaluating  the  contractor’s  performance  on  an  ongoing  basis  during  system 
development.  The  results  and  experience  that  the  contractors  gain  during  the  competitive 
fly-off  with  the  QAW  should  prepare  the  way  for  these  follow-on  software  architecture 


The  System  Evolution  Plan  is  a  contractor-developed  master  plan  describing  how  the  MSIS  will 
evolve  during  development  to  accommodate  the  transition  from  the  existing  legacy  systems  it 
will  replace. 

The  refinement  of  their  system  architecture  must  complement  their  approved  C4ISR 
operational  and  technical  architecture  baseline. 
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evaluations.  These  ATAM  evaluations  should  go  more  smoothly  and  provide  additional 
insight  into  risks  created  by  architectural  decisions,  find  trends  that  reveal  correlations 
between  architectural  decisions  and  predictions  of  system  properties,  and  surface  sensitivity 
points  and  tradeoffs  so  they  can  be  identified  and  documented  explicitly. 

5.7  Overview  of  the  Primary  QAW  Events 

Figure  5  provides  a  graphic  summary  of  the  approach  and  acquisition  artifacts  used  for 
integrating  and  applying  the  QAW  fi'om  the  issuance  of  the  RFP  through  the  initial  down 
select.  The  numbered  items  identify  the  sequence  in  which  the  QAW-related  events  occur 
and  associated  artifacts  are  generated.  The  lightly  shaded  ellipses  identify  the  first  two 
QAW  activities  that  the  DAC  is  responsible  for  performing.  And  the  call-out  box 
(number  5)  shows  the  QAW-related  artifact  that  the  offerors  are  responsible  for 
developing  and  delivering  to  the  DAC. 


Architecture 
Test  Cases 


ODmpetitive 

Solicitation 


initial 

Down  Select 


Competitive 

Fiy-Off 


Bidders  ^ 
Conferences 

RFP 


fMm 

iiCQn,6jad 
Awards;. 


Contractor  B  Performance 


Figure  5.  QAW  Integration  Approach  from  Issuance  of  the  RFP  Through  Initial 
Down  Select 

Similarly,  Figure  6  provides  a  graphic  summary  of  the  approach  and  acquisition  artifacts 
used  for  integrating  and  applying  the  QAW  from  the  award  of  the  competitive  fly-off 
contracts  through  the  final  down  select.  The  two  darkly  shaded  ellipses  (numbers  8  and 
11)  identify  the  QAW  activities  that  the  fly-off  contractors  are  responsible  for 
performing.  And  the  two  call-out  boxes  (numbers  7  and  10)  show  the  QAW-related 
artifacts  that  the  contractors  are  responsible  for  developing  and  delivering  to  the  DAC. 
Item  number  9  shows  the  ENs  the  DAC  will  issue  to  the  contractors,  as  needed,  after  the 
Analysis  Results  Presentation. 
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Figure  6.  QAW  Integration  Approach  from  Competitive  Fly-Off  Through  Final 
Down  Select 


5.8  Example  RFP  Language  for  the  QAW 

Tables  3  through  8  provide  an  overview  of  the  detailed  strategy  that  the  DAC  wrote  into 
the  RFP  with  the  SETs  assistance.  Because  the  acquisition  strategy  described  in  this  case 
study  is  very  typical  of  DoD  acquisitions  [Barbacci  00],  many  DoD  organizations  should 
be  able  to  adapt  the  approach.  The  adaptation  would  involve  analyzing  the  events 
(described  in  the  tables),  tailoring  them  to  the  specific  needs  of  the  organization,  and 
developing  the  corresponding  RFP/contract  language. 

Examples  of  the  QAW  RFP/contract  wording  for  the  ROE,  and  Sections  H,  L,  and  M  of 
the  RFP  are  included  in  Appendix  B  through  E,  respectively.  The  examples  provided  in 
the  appendices  for  Section  M  and  L  do  not  cover  all  aspects  of  the  final  down  select. 
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6  Accomplishments,  Lessons  Learned,  and 
Prognosis 


6.1  Accomplishments 

With  the  support  and  collaboration  of  key  MSIS  stakeholders  from  the  command, 
regional,  and  local  centers,  good  things  happened  in  planning  and  executing  this 
acquisition  initiative.  The  contribution  of  the  SEI  team  was  to  assist  the  DAC  in  planning 
and  devising  an  approach  for  integrating  a  QAW.  The  results  included 

•  identifying  the  business  drivers  and  quality  attributes  that  are  important  to 
stakeholders 

•  developing  an  approach  that  was  compatible  with  the  DAC’s  acquisition  objectives 
and  MSIS  acquisition  strategy 

•  conducting  scenario-generation  working  sessions  with  key  MSIS  stakeholders  during 
the  Planning  and  Preparation  phase 

•  completing  and  coordinating  ATC  development  and  producing  a  robust  set  of  ATCs 
for  inclusion  in  the  RFP  that  reflect  the  quality  attributes  of  interest 

•  incorporating  a  QAW  architecture  analysis  and  evaluation  objective  in  the  SOO  and 
developing  the  ROE  to  govern  the  process 

•  developing  the  special  contract  requirements  (Section  H)  for  conducting  a  QAW  and 
the  technical  evaluation  criteria  (Section  M)  governing  the  evaluation  of  QAW 
results 

•  developing  instructions  to  guide  offerors  in  preparing  the  QAW-related  aspects  of  the 
technical  proposal  (Section  L) 

•  creating  presentations  to  brief  potential  suppliers  about  the  QAW  at  the  upcoming 
bidder’s  conferences  as  well  as  DAC  management,  acquisition  olficials,  and  other 
stakeholders 

The  Scenario  Generation  and  Architecture  Test  Case  Development  activities  enabled  the 
DAC  to  establish  a  proactive  means  of  working  with  the  organizations  that  wilt 
eventually  be  using  the  MSIS.  Completing  these  activities  enabled  the  DAC  to 

•  better  understand  the  business  drivers  for  the  MSIS  and  what  quality  attributes  are 
most  important  to  the  stakeholders 

•  gauge  the  degree  to  which  stakeholders  and  legacy  contractors  share  a  common  view 
of  how  the  MSIS  should  operate 

•  identify  and  specify  scenarios  of  concern  to  the  stakeholders  and  the  architectural 
issues  and  concerns  associated  with  those  scenarios 

•  explore  how  the  C4ISR  architectural  views  should  be  supplemented  to  support 
architecture  analysis  and  evaluation 
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•  develop  a  set  of  ATCs  for  use  in  evaluating  the  ability  of  the  proposed  system  to 
achieve  the  desired  quality  attributes 

The  results  established  a  solid  “analysis  baseline”  that  the  DAC  and  MSIS  stakeholders 
can  build  on  during  the  MSIS  System  Implementation  phase.  This  will  assist  them  in 
using  the  ATAM  to  evaluate  the  software  architecture’s  inherent  quality  attribute 
sensitivities,  tradeoffs,  and  risks  early  in  the  System  Implementation  phase. 

From  the  DAC’s  standpoint,  the  bottom  line  was  that  all  the  participants  benefited  from 
the  experience  they  gained  in  preparing  for  and  integrating  the  QAW. 


6.2  Lessons  Learned 

During  the  Planning  and  Preparation  phase,  the  acquisition  strategy  changed  several 
times.  As  a  result,  the  SEI  team  had  to  modify  the  approach  for  integrating  the  QAW  and 
rewrite  sections  of  the  RFP  dealing  with  architecture  analysis  and  evaluation.  Developing 
a  robust  acquisition  strategy  should  be  one  of  the  highest  priority  items  in  the  early 
planning  phase  because  changing  the  strategy  downstream  can  cause  instability  and 
require  substantial  rework  of  the  RFP. 

Although  it  is  too  soon  to  gauge  the  ultimate  benefits  of  this  work,  the  issues  that 
surfaced  helped  the  DAC  to  discover  problems  early  in  the  acquisition  Planning  and 
Preparation  phase,  when  the  expense  of  correcting  them  is  substantially  less.  One 
example  is  that  the  original  acquisition  strategy  did  not  ensure  that  the  results  of  the 
competitive  fly-off  could  be  used  in  the  final  source  selection.  Another  example  is  that 
the  original  technical  evaluation  criteria  for  the  RFP  addressed  only  the  first  down  select. 

There  are  a  few  lessons  learned  that  are  relevant  to  other  acquisitions  desiring  to 
incorporate  architecture  analysis  and  evaluation.  Each  lesson  is  described  below  in  the 
form  of  a  problem  statement  and  resulting  lessons  learned. 

1.  How  could  the  DAC  require  an  offeror  to  plan  on  conducting  an  architecture  analysis 
using  a  specific  method  (such  as  the  QAW),  when,  under  the  tenets  of  acquisition 
reform,  acquirers  are  to  avoid  telling  potential  suppliers  what  to  do  and  how  to  do  it? 

Lessons  learned:  First,  acquisition  reform  allows,  and  even  encourages,  acquirers  to 
incorporate  risk-reduction  measures  in  an  acquisition.  As  a  result,  it  is  legitimate  for 
the  DAC  to  require  that  architecture  analysis  and  evaluation  be  incorporated  as  an 
acquisition  risk-reduction  measure,  since  architecture  plays  such  a  major  role  in 
determining  a  system’s  behavior.  Second,  the  DAC  must  ensure  that  there  is  a  “level 
playing  field”  when  evaluating  competing  architectures.  The  DAC  would  not  be  able 
to  equitably  evaluate  the  architecture  analysis  results  if  each  offeror  were  to  propose 
its  own  architecture  analysis  method  and  make  different  assumptions  about  the  scope 
and  depth  of  the  analysis  and  the  specific  architecture  aspects  it  would  evaluate. 
Specifying  a  QAW  provides  a  common  basis  for  planning  and  conducting  architecture 
analysis  and  evaluation  across  all  offerors. 

2.  By  policy,  the  DAC  was  required  to  use  a  SOO  instead  of  a  traditional  SOW  for  the 
MSIS  RFP.  This  means  that  each  offeror  is  responsible  for  developing  a  customized 
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sow  based  on  the  SOO  and  RFP  requirements.  The  first  question  this  policy  raises  is 
“what  is  an  appropriate  SOO  objective  to  include  in  the  RFP  to  accommodate 
architecture  analysis  and  evaluation  consistent  with  a  QAW?” 

Lessons  learned:  The  objective  used  read:  “Support  the  objective  of  reducing 
program  risk  by  participating  in  architecture  analysis  and  evaluation  for  the  life  of  the 
MSIS  acquisition.” 

The  second  question  this  policy  raises  is  “if  each  offeror  is  responsible  for  preparing 
the  SOW  that  describes  the  tasks  it  will  perform,  how  can  the  DAC  explicitly  specify 
in  the  RFP  that  the  offeror’s  approach  (and  hence  its  SOW)  must  include  performing 
an  architecture  analysis  using  the  QAW  approach?” 

Lessons  learned:  Use  Section  H  (Special  Contract  Requirements)  of  the  RFP  and 
include  a  common  set  of  requirements  to  govern  the  planning  and  conduct  of  the 
architecture  analysis  and  evaluation.  Specify  the  scope  of  the  architecture  analysis  and 
the  analysis  method  (e.g.,  QAW  or  ATAM)  in  the  Section  H  requirements. 


6.3  Prognosis 

As  a  result  of  integrating  the  QAW  in  the  MSIS  acquisition,  the  DAC  is  confident  that  it 

will  have  an  effective  means  to 

•  gauge  a  potential  supplier’s  ability  to  plan  and  conduct  an  architecture  analysis 

•  ensure  that  the  contractors  will  perform  a  thorough  architecture  analysis 
commensurate  with  the  architectural  views  and  documentation  that  are  available  at 
the  time 

•  ensure  that  the  architecture  analysis  results  will  provide  an  equitable  basis  for 
evaluating  how,  and  to  vyhat  extent,  the  contractor’s  proposed  architecture  can  or 
cannot  meet  the  specified  technical  requirements  and  system  quality  attributes 

•  set  the  stage  for  conducting  software  architecture  evaluations  downstream  using  the 
ATAM 

•  reduce  program  risk 
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7  Summary 


An  architecture  analysis  and  evaluation  is  one  risk  mitigation  activity  that  has  been 
shown  to  have  a  high  payoff.  Although  conducting  an  architecture  evaluation  may  appear 
to  be  an  obvious  step,  it  certainly  isn’t  a  routine  occurrence  in  the  DoD  and  government 
environment. 

We  have  presented  a  case  study  showing  how  the  SEI  was  able  to  assist  one  DoD 
organization  in  effectively  integrating  a  QAW  architecture  analysis  and  evaluation  in  its 
system  acquisition.  Integrating  architecture  analysis  and  evaluation  is  dependent  on 

•  the  goals  and  objectives  of  the  acquisition  program 

•  the  organization’s  acquisition  practices 

•  adapting  the  approach  (in  this  case,  the  QAW)  to  the  acquisition  strategy 

•  ensuring  that  essential  QAW  elements  and  supporting  artifacts  are  incorporated 
appropriately  into  the  RFP/con tract 

We  have  described  the  specific  acquisition  events  and  supporting  artifacts  needed  to 
incorporate  a  QAW  in  a  system  acquisition  so  that  architecture  analysis  can  play  a  major 
role  in  source  selection.  This  includes  providing  samples  of  the  contractual  language 
included  in  the  RFP  that  provide  insight  into  the  technical  implementation  details.  A  DoD 
organization  should  be  able  to  adapt  this  architecture  analysis  and  evaluation  approach  to 
its  specific  acquisition  needs  if  it  has  a  similar  acquisition  strategy  with  a  “rolling”  down 
select. 
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Feedback  and  Contact 


Comments  or  suggestions  about  this  document  or  the  series  of  technical  notes  on 
architecture  evaluation  in  the  DoD  are  welcome.  We  want  this  series  to  be  responsive  to 
the  needs  of  DoD  and  government  personnel.  To  that  end,  comments  concerning  this 
technical  note,  the  inclusion  of  other  topics,  or  any  other  issues  or  concerns  will  be  of 
great  value  in  continuing  this  series.  Comments  or  suggestions  should  be  sent  to 

Linda  Northrop,  Director 
Product  Line  Systems  Program 
lmn@sei.cmu.edu 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 


28 


CMU/SEI-2002-TN-013 


Appendix  A 


Example  QAW  Scenario  and  Architectural 
Test  Case 


Example  Scenario 

In  one  of  the  scenario-generation  brainstorming  sessions,  14  scenarios  were  generated, 
and  facsimiles  of  three  of  them  are  listed  below.  The  participants  then  grouped  these  three 
scenarios  together  as  representing  a  single  scenario. 

The  scenarios  included 

(I)  A  “Weapons  Platform  Generation  Program”  is  necessary,  which  is  tailorable  for 
each  separate  command. 

(II)  During  “contingency  generation”  a  simplified  information  flow  is  necessary, 
which:  avoids  duplication;  is  tailorable;  and  provides  read  access  to  users  at  the 
command  center  and  regional  centers. 

(13)  Must  provide  the  weapons  platform’s  status,  configuration,  locks,  and  parking 
location  to  the  weapons  platform's  crew  and  maintenance  supervisor.  This 
information  should  be  available  to  authorized  users  in  the  regional  and  command 
centers  on  a  need-to-know  basis. 

The  scenario  refinement  consists  of  building  the  outline  of  the  ATC  as  shown  below. 

First,  it  defines  the  context  and  the  organizations  involved,  and  then  lists  the  questions  to 
be  used  in  analyzing  the  architecture. 


Example  QAW  ATC 

The  following  example  is  an  ATC  that  was  developed  during  the  QAW  Test  Case 
Development  activity  from  the  initial  scenario  described  above.  The  participants  at  the 
brainstorming  session  reviewed  the  scenarios  and  created  an  operational  context 
described  below,  along  with  a  list  of  the  organizations  involved  and  the  quality  attributes 
of  interest. 
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1.1  Context 


A  unit  is  tasked  to  deploy  to  a  forward-operating  base  (contingency  generation, 
hurricane  evacuation,  etc.)  for  an  unknown  duration.  Key  leadership  has  a  need 
to  know  information  such  as  the  current  status,  the  location  and  mission-capable 
(MC)  rate,  the  long-range  scheduled  maintenance,  depot  inputs/outputs,  the 
current  status  of  the  hangar  queen,  and  the  available  hangar  space  for  aircraft 
being  generated  (prepared  bombs,  fuel,  etc.).  Information  will  be  used  to 
determine  which  aircraft  will  be  sent  and  in  what  order. 

1 .2  Quality  Attributes  of  Interest 

These  were  performance,  availability,  Interoperability,  security,  usability,  and 
modifiability.  The  utility  table  shows  which  quality  attributes  are  related  to  what 
questions. 

1.3  Questions 

1 .  How  will  the  system  generate  the  reports  (weapons  platform  status  report, 
fleet  health  report)  for  the  command  center  users  in  less  than  one  hour? 

Expected  Response.  Create  a  sequence  diagram  showing  the  distributed 
business  objects  used  In  generating  each  report.  Then  elaborate  on  this 
diagram  to  show  the  communication  delays  and  the  computational  times 
required  for  each  node.  The  timing  estimates  should  account  for  waiting 
times  for  both  normal  communication  traffic  and  access  to  computational 
nodes. 

2.  How  can  the  decision  makers  gain  access  to  the  above  information 
(securely  and  electronically),  without  having  to  use  a  paper  copy  from  the 
provider? 

Expected  Response.  Determine  alternative  ways  of  accessing  the  data 
from  various  devices,  enabling  interoperability  between  devices  and 
servers,  and  formatting  the  data  for  screen  presentation  and  browsing. 

3.  How  will  the  system  prevent  unauthorized  access  to  this  information  without 
impeding  access  by  the  maintenance  users  at  the  local  center? 

Expected  Response.  Create  a  delineation  of  the  security  software 
architecture. 

4.  How  will  the  system  prevent  the  loss  of  information  when  a  failure  occurs? 

Expected  Response.  The  architecture  should  show  how  data  Is 
maintained  when  a  failure  occurs,  In  either  redundant  components  or 
persistent  storage.  This  should  Include  a  discussion  of  access  to 
information  while  the  failure  persists. 

5.  How  will  the  system  support  24/7  operations  with  no  more  than  10  minutes 
of  downtime  in  a  24-hour  period? 

Expected  Response.  The  architecture  should  show  redundant  processes, 
servers,  and  communication  paths.  A  discussion  of  the  repair  strategy 
should  also  be  included.  A  reliability,  maintainability,  and  availability  (RMA) 
study  should  be  included. 
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6.  How  will  the  system  cross-check  or  automatically  correct  discrepancies  in 
the  data  (data-entry  verification)? 

Expected  Response.  Create  a  sequence  diagram  showing  how  a  complex 
data  entry  will  be  verified  at  different  levels  of  processing  within  the  system 
(e.g.,  at  a  data-entry  device,  a  local  server,  and  a  global  database  server). 

7.  How  will  the  system  highlight  a  pending  or  broken  suspense  (i.e., 
something  that  is  not  going  to  be  done  on  schedule)? 

Expected  Response.  Create  a  sequence  diagram  showing  the  impact  of  a 
broken  suspense,  and  describe  where  and  how  the  broken  suspense  is 
detected. 

8.  How  will  the  system  avoid  redundant  data  entry  (i.e.,  single  point  of  data 
entry,  for  example,  changing  a  weapon  platform’s  status  or  the  estimated 
time  of  completion  [ETIC])? 

Expected  Response.  Describe  how  a  single  entry  of  data  will  cause  all 
copies  of  that  data  to  be  updated.  Note:  This  could  be  difficult  to  do  before 
the  detailed  design  stage. 

9.  How  will  the  information  be  presented  to  the  end  user  In  a  simple,  easy-to- 
read,  and  easy-to-navigate  format  (e.g.,  colors,  alarms,  icons,  sounds)? 

Expected  Response.  Create  a  prototype  user  interface  demonstrating  the 
desired  capabilities. 

10.  How  can  the  system  help  tailor  command-  and  base-specific  munitions 
generation  tasks? 

Expected  Response.  Create  a  sequence  diagram  showing  how  the 
munitions  generation  task  can  be  tailored  to  develop  command-specific 
instances,  and  showing  how  these  tasks  can  be  further  tailored  to  develop 
base-specific  instances. 

1 1 .  Who  is  going  to  have  administrative  control  of  the  system  (e.g.,  installing, 
upgrading,  adding  users,  repairing)? 

Expected  Response.  Create  a  sequence  diagram  for  each  activity, 
including  the  objects  that  perform  the  installations/upgrades/repairs,  and 
the  online  objects  that  contain  instructions,  guidelines,  or  help. 

12.  How  will  the  system  assist  in  the  tailoring  of  the  system  to  meet  the  needs 
of  different  regional  centers? 

Expected  Response.  Describe  how  commonalities,  differences,  and 
variations  between  regional  center  operations  can  be  used  to  compose  a 
system. 
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1.4  Utility  Table 


Quality 

Attribute 

Quality  Attribute  Phrase 

Addressed 

in 

Question 

Performance 

“in  less  than  one  hour” 

1 

Interoperability 

“gain  access  electronically  to  the  report” 

2 

Security 

“prevent  unauthorized  access  to” 

3 

Availability 

“prevent  the  loss  of  information” 

4 

“24/7  operation  with  less  than  10 
minutes  of  downtime  per  day” 

5 

Usability 

“cross-check  or  automatically  correct 
discrepancies” 

6 

“highlight  a  pending  suspense” 

7 

“avoid  redundant  data  entry” 

8 

“easy-to-read  and  easy-to-navigate 
format” 

9 

Modifiability 

“tailor  command-generation  tasks” 

10 

“who  has  administrative  control  of  the 
system” 

11 

“tailor  the  system  to  meet  the  regional 
centers’  needs” 

12 
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Appendix  B 


Example  Rules  of  Engagement  for  the  QAW 


Note:  The  following  rules  of  engagement  are  relevant  to  conducting  the  QAW 
architecture  analysis  and  evaluation  events  described  in  the  MSIS  RFP/contract.  The 
language  that  is  provided  is  for  illustration  purposes  only  and  should  not  be  construed  as 
complete. 


1 .1  Contractual  Matters 

Any  adjustments  to  contract  requirements  must  be  made  through  the  Contracting 
Officer  and  adhere  to  established  contractual  practices.  If  issues  arise  In 
conducting  the  architecture  analysis  and  evaluation,  the  contract  shall  take 
precedence  followed  by  the  contractor’s  Government-approved  architecture 
analysis  plan. 

1.2  Technical  Interchange  Meetings 

Technical  Interchange  Meetings  (TIMs)  will  be  held  in  accordance  with  the  contract 
and  the  contractor’s  Government-approved  Architecture  Analysis  Plan  (AAP)  or  as 
approved  by  the  Contracting  Officer.  The  contractor  will  be  responsible  for 
scheduling  TIMs  for  each  QAW  event  or  task  that  involves  Government 
participation  or  oversight. 

1.3  Kick-Off  Meeting 

Following  the  award  of  the  Fly-Off  contracts,  the  Government  will  hold  a  separate 
kick-off  meeting  (i.e.,  a  special  TIM)  with  each  contractor  to  review  the  QAW 
process  and  Government  expectations  of  fly-off  contractors. 

1.4  Contractor  Participants  in  TIMs 

Contractor  personnel  participating  in  TIMs  or  other  scheduled  events  identified  in 
the  AAP  shall  include,  as  a  minimum,  the  AAP  team  leader,  the  chief  architect,  one 
or  more  domain  experts,  the  contractor’s  software  development  team  leader,  and 
representative  personnel  from  the  teams  responsible  for  performing  the 
architecture  analysis  and  developing  the  architectural  views  and  documentation. 
This  team  will  be  referred  to  as  the  contractor’s  architecture  development  team 
(ADT). 

1.5  Government  Participants  in  TIMs 

Government  personnel  participating  in  TIMs  or  other  scheduled  events  identified  in 
the  AAP  shall  include  a  mix  of  Government  personnel,  federally  funded  research 
and  development  center  (FFRDC)  personnel,  and  designated  contractors  who 
directly  support  the  MSIS  program  office.  This  team  will  be  referred  to  as  the 
Government’s  architecture  evaluation  team  (AET). _ 
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1 .6  Evaluation  Notices 

Following  each  dry-run  and  full-scale  QAW  Analysis  Results  Presentation,  the 
Contracting  Officer  shall  issue  Evaluation  Notices  (ENs),  as  appropriate,  on  behalf 
of  the  AET.  The  ENs  will  identify  any  items  needing  clarification  (e.g.,  anomalies, 
ambiguities,  and  issues)  or  any  deficiencies  requiring  resolution.  Deficiencies  cite 
conditions  that  will  not  satisfy  the  system  requirements  (including  failure  to  meet 
the  expected  response  for  test  cases)  or  conditions  that  represent  potential  risks  in 
terms  of  meeting  the  system  quality  requirements.  It  is  the  contractor’s 
responsibility  to  have  the  ADT  resolve  all  ENs. 

1 .7  Configuration  Management 

The  QAW  Test  Case  Architecture  Analysis  and  Analysis  Resuits  Presentation 
activities  shall  not  be  conducted  until  the  baseline  for  the  architecture  being 
analyzed  has  been  formally  established  and  entered  into  the  contractor's 
configuration  management  system. _ 


34 


CMU/SEI-2002-TN-013 


Appendix  C 


Section  H  of  the  RFP:  Speciai  QAW 
Requirements 


Note:  Included  in  Section  H  were  special  requirements  to  have  the  contractors  plan  and 
conduct  a  QAW-based  architecture  analysis.  These  requirements  are  described  below. 
The  language  that  is  provided  is  for  illustration  purposes  only  and  should  not  be 
construed  as  complete. 


Architecture  Analysis 

The  contractor  shall  conduct  a  Quality  Attribute  Workshop  (QAW)  as  a  risk 
reduction  measure  in  accordance  with  an  approved  plan  that  the  contractor  is 
responsible  for  developing  and  which  the  Government  must  approve.  The  special 
requirements  governing  the  planning  and  conduct  of  the  QAW-based  architecture 
analysis  are  described  in  the  following  sections. 

Architecture  Analysis  Plan 

The  offeror  shall  develop  an  Architecture  Analysis  Plan  (AAP)  for  conducting  a 
QAW.  The  plan^®  shall  Integrate  the  architecture  analysis  activities  and  events 
outlined  in  Attachment  A.^^  The  technical  aspects  of  the  AAP  shall  conform  to  the 
QAW  process^®  (i.e.,  principles  and  steps)  as  described  by  Barbacci  and 
associates  [Barbacci  02]  and  the  following  requirements. 

The  AAP  shall: 

a.  Describe  the  contractor’s  overall  approach  and  process  for  conducting  the 
QAW-based  architecture  analysis. 

b.  Provide  a  schedule  for  all  QAW  events  and  tasks  the  contractor  is  responsible 
for  performing  and  Technical  Interchange  Meetings  (TIMs)  with  the 
Government. 

c.  Identify  the  facilities  and  locations  where  the  contractor  will  conduct  the  TIMs, 

d.  Identify  all  needed  artifacts  and  resources,  including  what  architectural  views 
and  documentation  will  be  used  to  analyze  the  architecture  against  the 
Architecture  Test  Cases  (ATCs)  that  are  government-furnished  items  (GFIs). 

e.  Describe  how  the  architecture  analysis  results  will  be  recorded  and  presented. 

f.  Describe  how  deficiencies,  risks,  anomalies,  or  issues  discovered  during  the 
contractor’s  own  analysis  will  be  recorded,  tracked,  and  resolved. 

g.  Describe  how  any  Evaluation  Notices  (ENs)  that  the  Government  may  issue — 
in  response  to  the  contractor’s  walkthrough  and  presentation  of  the 
architecture  analysis  results — ^wlll  be  recorded,  tracked,  and  resolved. 

h.  Describe  how  the  architecture  will  be  updated  If  changes  are  made  by  the 
ADT. 


These  features  are  not  unique  to  planning  architecture  analyses;  they  are  characteristic  of  good 
planning  practices  and  planning  artifacts. 

Attachment  A  corresponds  to  the  events  and  tasks  outlined  in  Table  5  and  Table  6  of  this 
technical  note. 

Additional  information  is  available  on  the  SETs  Architecture  Tradeoff  Analysis  ( ATA) 
Initiative  Web  site:  <http://www.sei.cmu.edu/ata/ata_init.html>. 


CMU/SEI-2002-TN-013 


35 


All  offerors  will  submit  an  initial  AAP  with  their  technical  proposals.  The  contractors 
will  be  responsible  for  updating  their  AAPs  as  described  in  Sections  1  and  2  below. 

jhe  AAP  shall  Include  specific  provisions  for  (1)  conducting  a  dry-run  QAW  Test 
Case  Architecture  Analysis,  (2)  conducting  a  dry-run  QAW  Analysis  Results 
Presentation,  (3)  conducting  a  full-scale  QAW  Test  Case  Architecture  Analysis,  (4) 
conducting  a  full-scale  QAW  Analysis  Results  Presentation,  and  (5)  preparing  a 
report  describing  the  architecture  analysis  results  and  risk  mitigation  approach. 

Jhe  requirements  governing  these  5  elements  of  the  plan  and  some  ground  rules 
for  planning  and  conducting  the  QAW  are  described  in  more  detail  in  the  following 
sections. 

1.  Dry-Run  QAW  Test  Case  Architecture  Analysis 

After  the  contractor  has  updated  its  initial  AAP  (based  on  Government  comments 
from  the  Initial  review),  obtained  AAP  approval,  and  built  and  documented  a 
suitable  set  of  architectural  views,  the  contractor  shall  perform  a  dry-run 
architecture  analysis  in  accordance  with  the  tasks  and  schedule  of  events 
described  in  the  AAP.  This  initial  architecture  analysis  shall  mirror  the  architecture 
analysis  activity  described  in  Section  3  below,  only  the  analysis  shall  be  limited  to 
the  designated  set  of  ATCs^^  for  the  dry-run  that  will  be  provided  as  a  Government 
Furnished  Item  (GFI). 

The  purpose^^  of  the  architecture  analysis  is  two-fold:  (1)  to  identify  risks  and  (2)  to 
determine  the  extent  to  which  the  architecture  satisfies  the  ATCs  and  supports  the 
contractual  requirements  specified  in  the  MSIS  Technical  Requirements 
Specification  (TRS). 

2.  Dry-Run  QAW  Analysis  Results  Presentation 

After  the  contractor  has  analyzed  its  proposed  architecture  against  the  dry-run  set 
of  ATCs  identified  in  the  RFP,  the  contractor  and  Government  shall  jointly  conduct 
the  dry-run  QAW  Analysis  Results  Presentation.  The  presentation  shall  mirror  the 
full-scale  Analysis  Results  Presentation  activity  described  in  Section  4  below;  only 
it  shall  be  limited  to  the  dry-run  test  cases. 

The  objective  in  conducting  the  dry-run  QAW  Analysis  Results  Presentation  Is  to 
ensure  that 

a.  The  AAP  Is  adequate  for  conducting  the  follow-on,  full-scale  architecture 
analysis. 

b.  The  architectural  design  Is  suitably  documented  and  sufficiently  mature  to 
support  the  follow-on,  full-scale  analysis  of  the  contractor's  proposed 
architecture. 

c.  The  contractor  has  a  thorough  understanding  of  the  architecture  analysis 
method  and  a  demonstrated  ability  to  perform  the  analysis,  and  is  able  to 
review  and  present  the  analysis  results  in  a  reasonable  and  timely  fashion. 

d.  The  contractor  has  an  opportunity  to  resolve  any  issues  that  may  arise,  fine- 

tune  its  plan  and  proposed  architecture,  and  obtain  answers  to  any  questions  It 
might  have  about  the  analysis  method,  Its  application,  or  the  walkthrough  and 
presentation  of  the  analysis  results.  _ 


It  is  anticipated  that  after  contract  award,  the  contractor  may  want  to  update  its  initial  AAP  to 
include  refinements  that  reflect  new  information  or  knowledge  gained  during  the  time  the 
proposal  was  prepared  and  a  subsequent  contract  was  awarded. 

Three  architecture  test  cases  (one  each  for  the  MSIS  Command,  Regional,  and  Local  Centers) 
will  be  selected  by  the  Government  and  included  in  the  for  the  dry-run  architecture 

analysis. 

This  purpose  applies  to  both  the  dry-run  architecture  analysis  and  the  full-scale  architecture 
analysis  that  are  described  in  Sections  1  and  3,  respectively. 
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Following  the  dry-run  walkthrough  and  presentation  of  the  analysis  results,  the 
Government  will  issue  ENs  in  accordance  with  the  QAW  rules  of  engagement.  After 
resolving  any  ENs  Issued  by  the  Government,  the  contractor  shall  finalize  its 
architecture  analysis  plan  and  formally  submit  it  for  approval. 

3.  Full-Scale  QAW  Test  Case  Architecture  Analysis  (All  Test  Cases) 

After  the  dry-run  QAW  Analysis  Results  Presentation  is  complete,  and  the  AAP  has 
been  finalized  and  approved,  the  contractor  shall  complete  the  analysis  of  its 
proposed  MSIS  architecture  using  all  of  the  GFI  ATCs  in  accordance  with  the  tasks 
and  schedule  of  events  described  in  its  AAP.  The  full  set  of  GFI  ATCs  is  included  in 
the  RFP.  The  ATCs  are  designed  to  exercise  key  aspects  of  the  system  and  its 
architecture  to  assist  the  AET  in  determining  the  degree  to  which  the  system  quality 
requirements  and  other  related  system  requirements  are  or  will  be  satisfied.  Each 
architecture  test  case  will  identify  the  system  quality  attributes  of  interest  and 
reference  the  system  requirements  (specified  in  the  TRS)  that  apply  to  the  test 
case  analysis. 

Like  the  dry-run  architecture  analysis  that  preceded  the  dry-run  QAW  Analysis 
Results  Presentation,  the  full-scale  architecture  analysis  for  all  the  ATCs  shall  be 
conducted  in  accordance  with  the  QAW  Test  Case  Architecture  Analysis  activity 
described  by  Barbacci  and  associates  [Barbacci  02]  and  the  QAW  workbooks  that 
are  GFIs.  The  contractor  will  record  the  analysis  results  including  any  discovered 
deficiencies,  risks,  tradeoff  points,  anomalies,  or  issues.  Any  deficiencies,  risks, 
anomalies,  or  issues  will  be  entered  into  the  contractor’s  tracking/corrective  action 
system  along  with  plans  for  resolution  or  mitigation.  The  contractor  shall  resolve  all 
anomalies  and  issues  prior  to  producing  the  final  architecture  analysis  report. 
Following  the  analysis  and  resolution  of  anomalies  and  Issues,  and  the  mitigation 
(planned  or  actual)  of  deficiencies  and  risks  identified  during  the  analysis,  the 
contractor  shall  place  the  architecture  under  configuration  control,  using  the 
contractor’s  configuration  management  system.  Any  modifications  to  contractor 
work  efforts  resulting  from  this  analysis  will  be  entered  into  the  contractor’s  effort¬ 
tracking  system. 

4.  Full-Scale  QAW  Analysis  Results  Presentation  (All  Test  Cases) 

After  the  full-scale  architecture  analysis  has  been  completed  for  all  the  ATCs,  the 
contractor  and  Government  shall  jointly  conduct  the  full-scale  QAW  Analysis 
Results  Presentation  in  accordance  with  the  contractor’s  approved  AAP.  As  an 
adjunct  to  reviewing  the  analysis  results,  the  Government  will  also  evaluate  the 
contractor’s  progress  against  the  AAP.  The  Government  shall  be  afforded  ample 
opportunity  to  review  and  discuss  the  results  and  the  contractor  should  be  prepared 
to  discuss  the  analysis  results  In  detail  including  the  underlying  design  rationale, 
assumptions,  and  related  issues,  findings,  and  observations.  The  information  that  is 
presented  during  the  review  of  the  analysis  results  shall,  as  a  minimum,  correspond 
to  the  artifacts  and  information  identified  in  the  QAW  workbooks  that  are  GFIs.  The 
Government  will  issue  ENs  in  response  to  the  walkthrough  and  presentation  of  the 
analysis  results  in  accordance  with  the  QAW  rules  of  engagement. 

5.  Architecture  Analysis  Report  (AAR) 

Following  the  walkthrough  and  presentation  of  results,  the  contractor  shall  produce 
and  deliver  an  Architecture  Analysis  Report  (AAR)  in  accordance  with  the 
contractor’s  approved  AAP.  The  AAR  shall  Include  the  analysis  results  that  were 
presented  during  the  full-scale  QAW  Analysis  Results  Presentation  described  in 
Section  4  above.  The  report  shall  also  include  the  architectural  views  and 
documentation  that  were  used  in  analyzing  the  ATCs.  The  report  shall  document 
the  architecture  analysis  results  including  any  deficiencies,  risks,  anomalies,  and 
issues,  and  how  they  were  resolved.  In  addition,  the  report  shall  describe  the 
contractor’s  resolution  of  all  evaluation  notices  issued  by  the  Government  including 
the  actual  (or  planned)  approach  for  mitigating  any  cited  deficiencies  and  risks.  If 
subsequent  changes  are  made  to  the  architecture,  updated  architectural  views  and 
associated  documentation  shall  be  included  in  the  report  and  placed  under 
configuration  control,  using  the  contractor’s  configuration  management  system.  The 
analysis  results  included  in  the  AAR  shall  correspond  to  the  artifacts  and 
information  identified  in  the  QAW  workbooks  that  are  GFIs. 
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Appendix  D 


Section  L  of  the  RFP:  Offerors’  Instructions 


for  the  QAW 


Note:  Section  L  covers  instructions  to  offerors.  An  excerpt  of  the  instructions  that  pertain 
to  planning  and  conducting  a  QAW-based  architecture  analysis  is  included  below.  The 
language  that  is  provided  is  for  illustration  purposes  only  and  should  not  be  construed  as 
complete.  It  must  be  adapted  for  each  acquisition,  especially  in  light  of  each  acquisition 
organization’s  regulations,  policies,  and  guidance  for  source  selection  and  for 
procurement  actions. 


General 

In  Section  L  of  the  RFP,  the  acquirer  specifies  that  the  supplier  is  to  submit  a  cost 

proposal  and  a  technical  proposal  that  includes 

•  a  Technical  volume,  describing  how  the  offeror  will  conduct  a  QAW-based 
architecture  analysis  and  integrate  it  into  its  development  effort 

•  a  Past  Performance  volume,  describing  the  offeror’s  previous  work  in 
architecture-based  development  and  its  experience  in  software  architecture 
analysis  and  evaluation 

•  a  Pre-Award  Demonstration  volume,  in  which  the  offeror  describes  its  plans 
and  procedures  for  demonstrating  its  architecture-based  development 
capabilities  and  its  architecture  analysis  tools  and  skills 


QAW-Related  Instructions  Applicable  to  the  Technical  Volume 

1.  Initial  Technical  Proposal 

In  conjunction  with  the  offeror’s  initial  technical  proposal,  the  offeror  shall  produce 
an  Architecture  Analysis  Plan  (AAP)  describing  how  it  will  conduct  a  QAW-based 
architecture  analysis  in  accordance  with  the  Special  Contract  Requirements  of 
Section  H.  The  offeror  shall  provide  sufficient  details  and  substantive  information  in 
the  AAP  to  convey  to  the  evaluator  a  clear  and  accurate  understanding  of  its 
approach  for  conducting  a  QAW-based  architecture  analysis  and  to  permit  a 
complete  and  accurate  evaluation  of  the  AAP.  The  AAP  shall  also  identify  all  risks, 
uncertainties,  or  major  problems  in  meeting  the  technical,  delivery,  and  related 
requirements  and  quality  objectives  for  conducting  a  QAW-based  architecture 
analysis.  The  proposed  mitigation  of  these  risks,  uncertainties,  or  problems  shall 
also  be  described  in  the  plan. 

2.  Improved  Technical  Proposal 

In  conjunction  with  the  offeror’s  improved  technical  proposal  that  is  to  be  submitted 
in  response  to  the  Government’s  Call  for  Improvement  (CFI),  the  offeror  shall 
include  the  following  plans  and  reports. _ 
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2.1  Architecture  Analysis  Plan  (AAP),  Final  Version 

The  offeror  shall  include  the  final  version  of  the  AAP,  which  was  approved  by  the 
Government,  as  part  of  the  improved  technical  proposal  submitted  in  response  to 
the  CFI.  The  requirements  for  developing  the  AAP  are  described  In  Section  H  of 
the  RFP. 

2.2  Architecture  Analysis  Report  (AAR) 

The  offeror  shall  include  the  Architecture  Analysis  Report,  which  was  developed 
during  the  Fly-Off,  as  part  of  the  improved  technical  proposal.  The  requirements  for 
developing  the  AAR  are  specified  in  Section  H  of  the  RFP. _ 
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Appendix  E 


Section  M  of  the  RFP:  QAW  Technical 


Evaluation  Criteria 


Note:  The  language  in  this  section  represents  one  possible  way  of  requesting  areas 
related  to  architecture  analysis  and  evaluation  requirements  for  source  selection.  The 
language  that  is  provided  is  for  illustration  purposes  only  and  should  not  be  construed  as 
complete.  It  does  not  cover  all  aspects  of  source  selection  for  the  final  down  select.  The 
language  must  be  customized  for  each  acquisition  to  reflect  the  fact  that  the  QAW  is 
being  performed  in  a  broader  context  of  a  system  acquisition  and  that  the  evaluation 
factors  will  vary  with  each  acquisition  organization's  regulations,  policies,  and  guidance 
for  source  selection  and  for  procurement  actions. 


Example  Factors  And  Subfactors  To  Be  Evaluated 

The  following  factors  and  subfactors  will  be  evaluated. 

1 .0  Technical  Capability  Factor 

Note:  All  the  evaluation  factors  for  the  QAW-based  architecture  analysis  are 
included  under  Paragraph  1.1.2  below. 

1.1  Subfactor  1,  Architecture  Approach 

1.1.1  Evolutionary  Strategy 

1.1.2  Architecture  Analysis 

The  Government  will  evaluate:  (1)  the  contractor's  ability  to  plan  and  conduct  a 
QAW-based  architecture  analysis  and  (2)  the  analysis  results.  The  specific 
evaluation  criteria  that  will  be  used  to  evaluate  the  AAP  and  AAR  are  described 
In  the  following  sections. 

1.1. 2.1  Architecture  Analysis  Plan  (AAP) 

During  the  Initial  Down  Select,  each  offeror’s  Architecture  Analysis  Plan  (AAP) 
will  be  evaluated  for  the  adequacy  of  response  for  conducting  a  QAW-based 
architecture  analysis.  Adequacy  of  response  is  defined  as  the  extent  to  which  the 
Architecture  Analysis  Plan  (AAP)  is  complete  and  demonstrates  an 
understanding  of  the  QAW  architecture  analysis  requirements.  Completeness  is 
defined  in  terms  of  two  elements.  The  first  element  is  the  extent  to  which  the 
description  of  the  offeror’s  proposal  and  AAP  contains  sufficient  and  substantive 
information  to  convey  to  the  evaluator  a  clear  and  accurate  description  of  how 
the  QAW-based  architecture  analysis  requirements  will  be  satisfied.  And  the 
second  element  is  the  extent  to  which  the  proposal  and  plan  describe 
approaches  and  proposed  solutions  that  address  all  requirements  and 
associated  risks,  problems,  and  uncertainties  and  the  means  of  resolving  them. 
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The  evaluation  will  assess  the  adequacy  of  response  of  the  offeror’s  AAP 
including: 


22 


a.  Completeness  of  the  AAP  and  conformance  with  the  architecture  analysis 
activities  and  events  outlined  in  Attachment  A 

b.  Clarity  of  the  offeror’s  description  of  its  QAW-based  architecture  analysis 
planning  approach 

c.  Consistency  with  the  offeror’s  Initial  program  management  plan,  system 
evolution  plan,  and  spiral  development  plan 

d.  Feasibility  of  the  AAP  given  the  schedule  and  technical  requirements  stated 
in  the  RFP 

e.  Reasonableness  and  validity  of  the  planning  assumptions  identified  in  the 
AAP 

f.  Accuracy  and  fidelity  of  the  AAP  In  applying  the  QAW  architecture  analysis 
method 

g.  Adequacy  of  the  AAP  to  satisfy  the  special  contract  requirements  of  Section 
H  for  the  QAW-based  architecture  analysis 

1.1 .2.2  Architecture  Analysis  Results  (AAR) 

The  evaluation  of  the  offeror’s  Improved  Technical  Proposal  will  assess  the 
results  of  the  architecture  analysis  Including: 

a.  Ability  to  satisfactorily  complete  all  QAW-based  architecture  analysis  tasks 
and  events 

b.  Ability  to  satisfactorily  resolve  all  Evaluation  Notices  (ENs)  issued  by 
Government. 

c.  Feasibility  of  proposed  architecture  approach  to  meet  MSIS  requirements 
and  contractor’s  claimed  MSIS  capabilities 

d.  Any  Instances  of  architecture  non-compliance  with  system  and  architecture 
requirements  specified  in  the  Technical  Requirements  Specification  (TRS) 

e.  Risk  of  architectural  design  to  satisfy  MSIS  system  quality  attribute 
requirements  based  on  test  case  results 

f.  Ability  to  satisfy  expected  responses  specified  for  each  element  of  the 
architecture  test  cases 

g.  Demonstrated  capability  of  contractor  to  understand  MSIS  architecture  test 
cases  and  perform  analysis 

h.  Demonstrated  ability  to  execute  architecture  analysis  in  conformance  with 
approved  AAP 

i.  Clarity  and  completeness  of  architectural  views  and  documentation 

j.  Ability  of  architectural  views  and  documentation  to  support  each  element  of 
the  test  case  analyses 

k.  Delivery  of  all  revised  and  new  architectural  views  and  documentation 
needed  to  support  the  resolution  of  ENs 

l.  Clear  and  explicit  identification  of  any  architectural  changes  made  after  the 
full-scale  Analysis  Results  Presentation  or  incorporated  in  the  improved 
technical  proposal  including: 

1 .  Succinct  description  of  each  change 

2.  Supporting  rationale  for  each  architectural  change 

3.  Suitable  description  of  system  impact  and  efficacy  of  each  change 

4.  Clear  description  of  impact  of  each  change  on  individual  test  case 
results 


Attachment  A  corresponds  to  the  events  and  tasks  outlined  in  Table  5  and  Table  6  on  pages  18 
and  19,  respectively. 
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m.  Risk  of  architectural  changes  made  after  full-scale  Analysis  Results 
Presentation  introducing  conditions  that  may  jeopardize  satisfying  the  MSIS 
system  requirements,  achieving  the  MSIS  quality  attribute  requirements,  or 
satisfying  expected  response  for  the  GFI  test  case  results 

n.  Consistency  of  architectural  views  and  documentation  with  all  architectural 

changes  described  in  AAR  _ 
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